The Microsoft® Windows and Macintosh® operating systems have revolutionized computer usability for the sighted population with their graphical user interfaces. The graphical, window-based interface provides an intuitive, visual replacement for the complex internal representation of the underlying computer data. The vast majority of people find the visually-based organization much easier to use, and many years of development effort have been devoted to creating an easy-to-use environment.
A primary type of the graphical user interface utilizes active portions of a computer screen called windows. (These functional windows are not to be confused with the brand of software distributed by Microsoft that uses windows, which is confusingly called Microsoft Windows. Throughout this specification, functional portions of the computer screen will be referred to as windows (lower case “w”) and the brand of software distributed by Microsoft will be referred to as Microsoft Windows (brand name with uppercase “W”)). Many other such windows interfaces have been defined for computers, hand-held devices, game controllers, telephones and other electronic devices.
For the majority of the visually impaired users, however, the windows interface has made computer use more difficult. Experience has shown that most visually impaired users take much longer to learn to use windows software applications than they previously did for command line software applications, such as DOS. As described below, the current input/output technologies available to the visually impaired place them at a disadvantage with regard to computer usability. Furthermore, there are a growing number of situations where computers are used but associated computer monitors are not, such as the case in which space limitations preclude a computer screen on the monitor that is large enough to be useful. Therefore, it would be desirable to have a device that enables the user to navigate a graphical user interface by touch, With or without a computer monitor.
All windows software is of a visual, spatial nature, which puts the visually impaired at a significant disadvantage. Many aspects of the computer screen are experienced in parallel by the sighted person, who graphically controls the application with pointing and clicking motions. The user often returns to previously-selected objects on the screen, which can rapidly be accessed through sight. Each new generation of software is more graphically-oriented than the previous, and text is often used more as a hint to the function of the underlying command than the complete identifier.
To make windows software usable and understandable in circumstances where a computer screen cannot be seen or a monitor is not used, therefore, it is important for the interface to be spatially-oriented, to capture the parallel aspects of the software interface, and to allow the user to rapidly examine the available graphical options, and to be able to access previous locations through spatial, rather than memory techniques. Any approach that allows only a small part of the screen to be experienced at any point in time works against this goal. As discussed below, nearly all current approaches do just that.
Screen readers convert text in the window into speech. Various commands can be issued from the keyboard (or using a speech-understanding software), which cause menus to be read out, items to be selected, etc. Unfortunately, screen readers do not present a spatial or parallel experience of the screen. They tend to provide a confusing, lengthy listing of many irrelevant details, leaving the uninitiated user in a state of total confusion. Screen magnifiers magnify a small part of the screen to a size that is readable by the user. For many individuals, those who must magnify individual words or icons to the size of the screen, the spatial, parallel nature of windows software is also lost. The user must serially magnify many objects on the screen to reach the desired point. Unless an accurate mental model of the whole window or screen is established, it is very easy for these users to become hopelessly lost.
Another approach to windows interfacing is to present audible signals to the user to indicate graphical information. For example, U.S. Pat. Nos. 5,223,828 and 5,374,924 disclose technology to provide graphical information about when the mouse pointer is moved over window boundaries, when it is in the title area, etc. This helps to provide a measure of information to the user that is not available in speech-only approaches. The drawbacks to the audible signals are similar to the screen magnifier approach, in that the information is from only a small part of the screen. For the user to obtain a more global experience of the screen, the mouse must be scanned over the entire area. Going back to a previously-visited spot is difficult because no global, parallel experience of the screen is available.
Several approaches have been developed to provide force feedback or other tactile information to the user, such as those disclosed in U.S. Pat. Nos. 5,714,978; 5,736,978; 5,903,456; and 6,078,308. For example, the force required to move the mouse over boundaries can become greater as the distance to the boundary becomes less. Bumps and other tactile information can be presented to the user through this technology as the pointer is moved over various objects on the screen. Several researchers have developed technologies to present a tactile version of what is on the screen. U.S. Pat. No. 5,912,660 discloses using an array of pins to represent a tactile version of an image or part of an image, such as an array of 32 rounded pins for each of three fingers on the mouse. This approach is relatively complex and expensive, due to the mechanical nature of the output mechanism. Like the audible feedback approaches, these devices have the disadvantages of not providing a parallel, spatial sensation to the user.
A touch screen allows tactile input, but indicates the functional areas only visually and does not provide tactile output. Touch screens do not enable high-speed, multi-fingered operation from the user. Touch Graphics (www.touchgraphics.com) has developed a series of products in which thin, tactile surfaces, called overlays, for particular games and learning applications are placed over a touch screen. This approach provides a spatial, parallel tactile input surface. Unfortunately, however, typical windows applications rapidly change the information on the screen, such as by providing pop-up dialog boxes, drop-down menus, and other short-term graphical structures. It would be impractical to provide overlays for all the potentially-visible graphical structures in rapid succession. Additionally, a touch screen is a very costly item compared to other potential input technologies.
An interesting new technology applicable to a touchable screen is electro-rheological fluids, as disclosed in U.S. Pat. No. 5,222,895. When subjected to an electric field, the viscosity of the fluid changes quickly. A dynamically-changing tactile screen can therefore be created, by producing localized fields at the points where a raised sensation is desired. However, the technology for doing this at high resolution is currently very cumbersome and expensive.
To summarize, many technologies are available for providing a non-visual interface to windows software applications. However, they all have one or more serious shortcomings, the user cannot experience the entire window in parallel; or the interfaces are not adaptive to the typical dynamically-changing screens of windows software applications; or they are very expensive.
Therefore, it is an object of this invention to provide an interface that enables user to examine the contents of an entire window rapidly, and in parallel, by touch. It is another object of this invention to provide a tactile interface that is adaptive to the typical dynamically-changing screens of windows software applications. It is a further object of this invention to provide a tactile interface usable with all software applications having graphical user interfaces. It is another object to provide a tactile interface having low-cost, rugged technology.